A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms

This paper presents a database of " ground truth " segmentations produced by humans for images of a wide variety of natural scenes. We define an error measure which quantifies the consistency between segmentations of differing granularities and find that different human segmentations of the same image are highly consistent. One of many uses of this dataset is demonstrated in evaluating the performance of segmentation algorithms. 1 Two central problems in vision are image segmentation and recognition. Though we may argue that they are aspects of the same problems, there is one practical difference that concerns us: It is considerably easier to quantify the performance of recognition algorithms than segmentation algorithms. Recognition is classification. The ready availability of datasets such as MNIST for handwritten digits and FERET for faces has enabled different researchers to compare different approaches on a firm quantitative footing. In contrast, the evaluation of image segmentation algorithms remains largely subjective. Typically, researchers show results on a few images and explain why the results " look good ". This is despite the fact that there has been significant effort to improve the situation. Both Yang et al. [6] and Hoover et al. [2] establish ground-truth segmentations of real images by combining the manual segmentations of multiple domain experts. Heath et al. [1] evaluate edge detectors using human subjects to subjectively judge the recognizability of objects in the edge images. Mao et al. [3] develop a solid evaluation methodology using a 1000-image dataset. Unfortunately, these domain-specific datasets are not applicable to the problem at hand. Hoover et al. [2] calls for the community to create a large segmentation dataset using human segmentations as ground truth. The closest dataset we know of is the Sowerby dataset that has been used by several researchers to study the statistics of natural images. However, this dataset is small, not publicly available, and contains only one segmentation per image. This paper presents a new dataset of human-segmented natural images that will be made public. We chose 1000 representative images from the 40,000-image Corel image database that is widely used in computer vision. Our goal is to collect 4 color and 4 grayscale segmentations by different people of each of the 1000 images. The results presented here use 150 grayscale segmentations by 10 people of 50 images. The data collection is ongoing, and at this time, we have 4200 segmentations by 25 people 1 For …